CLAIMS 



1. An iterative method for creating and evaluating strategies, comprising the 
5 steps of: 

providing any of: 

a team development module for developing a strategy modeling team; 
a strategy situation analysis module for framing a decision situation; 
a data request and reception module for designing and executing logistics 
10 of specifying, acquiring, and loading data required for decision and strategy 
modeling; 

a data transformation and cleansing module for verifying, cleansing, and 
transforming data; 

a decision key and intermediate variable creation module for computing 
15 additional variables from data and constructing a data dictionary; 

a data exploration module for determining characteristics that are effective 
decision keys and intermediate variables; 

a decision model structuring module for formalizing relationships between 
decisions, decision keys, intermediate variables, and value of a decision model; 
20 a decision model quantification module for encoding information into a 

decision model; 

a strategy creation module for determining strategies that a client can test; 

and 

a strategy testing module for testing strategies to guide refinement of 
25 strategies and refinement of a decision model and to select a best strategy for 
deployment; 
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wherein each of said modules has capability to interact with an expert task 
manager, wherein said expert task manager provides expert knowledge about 
strategy modeling processes and sub-processes. 

5 2. The iterative method of Claim 1 , the step of providing said team development 
module further comprising: 

said strategy modeling team executing analysis to allow a leader of said 
strategy modeling team to convince a decision maker to implement a strategy 
favored by said analysis. 

10 

3. The iterative method of Claim 1 , the step of providing said strategy situation 
analysis module further comprising: 

identifying the values of the organization; and 

ensuring that the right decisions and strategies are considered in an analysis. 

15 

4. The iterative method of Claim 1 , the step of providing said data request and 
reception module further comprising: 

designing and executing logistics of specifying, acquiring, and loading data 
required for decision and strategy modeling. 

20 

5. The iterative method of Claim 1 , the step of providing said data transformation 
and cleansing module further comprising: 

verifying, cleansing, and transforming data. 

25 6. The iterative method of Claim 1 , the step of providing said decision key and 
intermediate variable creation further comprising: 
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computing intermediate variables from said data, said intermediate variables 
dependent on decision keys; and 

constructing a data dictionary. 

7. The iterative method of Claim 1 , the step of providing said data exploration 
module further comprising: 

providing insight into said data by determining which decision keys are most 
relevant for predicting said intermediate variables; and 

gaining insight into a customer's business and business processes. 

8. The iterative method of Claim 1 , the step of providing said decision model 
structuring module further comprising: 

formalizing relationships between decisions, decision keys, intermediate 
variables, and value by connecting such in a model. 

9. The iterative method of Claim 1 , the step of providing said decision model 
quantification module further comprising: 

encoding information into a decision model. 

10. The iterative method of Claim 1, the step of providing said strategy creation 
module further comprising: 

applying optimization methods to a decision model to determine an optimal 
strategy for a set of cases. 

1 1 . The iterative method of Claim 1 , the step of providing said strategy creation 
module further comprising: 
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evolving using results from a decision model being enriched and from 
strategies tested. 

12. The iterative method of Claim 1, the step of providing said strategy testing 
5 module further comprising: 

providing means for evaluating each strategy based on simulation; and 
providing means for evaluating a strategy in the field. 

13. The iterative method of Claim 1 , further comprising the steps of: 

10 beginning with a simplified value model having less than eight drivers; 

wherein each of said drivers is modeled crudely by one or two decision keys; 
initially including no constraints; 

using said simplified value model for beginning said strategy creation module 
and said strategy testing module, said strategy creation module and said strategy 
15 testing module indicating areas of said decision model where refinement adds 
particular value; and 

after interaction between said decision model and strategies is acceptable, 
iteratively adding details reflecting limitations of a business process. 

20 14. The iterative method of Claim 1, wherein said team development module 
comprises a team creation component and a decision quality component. 

15. The iterative method of Claim 1 , further comprising the step of: 

providing a decision quality process for enabling an organization to 
25 systematically identify, understand, and track views of quality of decision making. 



198 



1 6. The iterative method of Claim 1 , further comprising the step of: 

providing any of six dimensions associated with any of six links in a decision 
quality chain, said any of six links comprising: 
appropriate frame; 
5 creative-feasible alternatives; 

meaningful-reliable Information; 
clear values and tradeoffs; 
logically-correct reasoning; and 
commitment to action; 
10 wherein said chain supports an organization's value. 



17. The iterative method of Claim 1, said step of providing a strategy situation 
analysis module further comprising the steps of: 
framing a problem by: 
15 identifying issues; 

developing a decision hierarchy; 
understanding an organization's values; and 
brainstorming and clarifying alternatives; 
further understanding said organization's values by: 
20 developing value metrics and prototyping metric results; and 

planning for data acquisition by: 

identifying intermediate variables; and 
developing a plan for assessment; 
wherein for clarification: 
25 optionally returning to said framing a problem step after said further 

understanding said organization's values step; and 
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optionally returning to said further understanding said organization's 
values step after said planning for data acquisition step. 

18. The iterative method of Claim 1, the step of providing said data request and 
5 reception module further comprising the steps of: 

developing data parameters, including: 

determining data elements; 

designing a performance period; 

determining data records; and 
10 constructing an initial data dictionary; 

determining transfer parameters, including: 

determining transfer format; and 

determining transfer method; 
preparing data, including: 
15 assembling transfer data; and 

transferring data; and 
loading data on a target system. 

19. The iterative method of Claim 1, said step of providing a data transformation 
20 and cleansing module further comprising the steps of: 

validating original data sets, comprising: 

investigating original data sets; and 

cleaning original data sets; 
creating analysis data sets, comprising; and 
25 transforming data; and 

computing additional variables; 
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validating analysis data sets, comprising; 
transforming data; and 
computing additional variables; 
wherein while creating analysis data sets and problems are uncovered in 
5 original data sets, then original data sets are further cleaned and retransformed; and 
wherein while validating analysis data sets and problems in said 
transformation, or in original data sets, are uncovered, then such tasks are revisited. 



20. The iterative method of Claim 1 , said step of providing a decision key and 
10 intermediate variable creation module further comprising the steps of: 

first creating dependent variables useful for decision models, comprising: 
identifying concepts; 
triaging concepts; and 
defining dependent variables; and 
15 creating independent variables useful for decision models, comprising 

identifying concepts; 
triaging concepts; and 
defining dependent variables; 
wherein intermediate variables depend on decision keys, other intermediate 
20 variables, or decisions; and 

wherein each intermediate variable encapsulates a predictive model with a 
dependent variable and independent variables. 

21. The iterative method of Claim 1, said step of providing a data exploration 
25 module further comprising the steps of: 

applying basic statistical analysis, comprising: 



201 



analyzing continuous variables; and 

analyzing discrete variables; 
applying variable reduction techniques, comprising: 

applying human and business judgment; and 
5 applying computational methods; 

applying advanced statistical analysis; 
verifying results; and 
presenting said results. 



10 22. The iterative method of Claim 1, said step of providing a decision model 
structuring module further comprising the steps of: 
conceptualizing, comprising the steps of: 

selecting intermediate variables that drive value; 
building coarse models of intermediate variables; and 
15 verifying constraints; and 

drawing a decision model structure; 

wherein said conceptualizing step is iteratively available for use after said 
drawing step. 

20 23. The iterative method of Claim 1 , said step of providing a decision model 
quantification module further comprising the steps of: 
modeling intermediate variables; 

filling in nodes with models, functions, and/or constants; and 
validating said decision model; 
25 wherein said modeling step is iteratively available from said filling in step, and 

wherein said filling in step is iteratively available from said validating said decision 
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model step. 

24. The iterative method of Claim 1, further comprising the step of providing a 
score tuner component for automating decision model updating and reporting, said 
5 score tuner component comprising any of: 
data awareness capability; 
triggering rules; 
model history retention; 
self-guided model development; 
10 connection to a decision engine; and 

execution and analytic audit trails; 

wherein when a tuning run is triggered, results are reviewed and either 
accepted and an update is deployed, or rejected. 

15 25. The iterative method of Claim 1, said step of providing a strategy creation 
module further comprising the steps of: 

performing model optimization, comprising: 
identifying metric variables; 
determining optimization parameters; and 
20 running optimization; 

analyzing optimization results, comprising 
viewing optimization results; and 
performing sensitivity analysis on constraints; and 
developing strategies, comprising: 
25 building strategies; and 

refining strategies; 
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wherein the performing model optimization step and the analyzing 
optimization results step are available to be used iteratively from either the analyzing 
optimization results step or the developing strategies step. 



5 26. The iterative method of Claim 1 , further comprising the step of: 

providing a non-linear constrained optimization tool for improving test designs 
and optimizing strategies. 



27. The iterative method of Claim 1 , said step of providing a strategy testing 
10 module further comprising the steps of: 
testing strategies, comprising: 

performing strategy simulation; and 
performing field testing; 
evaluating strategies; and 
15 performing active data collection; 

wherein said testing strategies step is available for being used iteratively from 
said evaluating strategies step. 



28. An apparatus for iteratively creating and evaluating strategies in an iterative, 
20 comprising: 

means for providing any of: 

a team development module for developing a strategy modeling team; 
a strategy situation analysis module for framing a decision situation; 
a data request and reception module for designing and executing logistics 
25 of specifying, acquiring, and loading data required for decision and strategy 
modeling; 
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a data transformation and cleansing module for verifying, cleansing, and 
transforming data; 

a decision key and intermediate variable creation module for computing 
additional variables from data and constructing a data dictionary; 
5 a data exploration module for determining characteristics that are effective 

decision keys and intermediate variables; 

a decision model structuring module for formalizing relationships between 
decisions, decision keys, intermediate variables, and value of a decision model; 

a decision model quantification module for encoding information into a 

10 decision model; 

a strategy creation module for determining strategies that a client can test; 

and 

a strategy testing module for testing strategies to guide refinement of 
strategies and refinement of a decision model and to select a best strategy for 
15 deployment; 

wherein each of said modules has capability to interact with an expert task 
manager, wherein said expert task manager provides expert knowledge about 
strategy modeling processes and sub-processes. 

20 29. The apparatus of Claim 28, said team development module further 
comprising: 

means for said strategy modeling team executing analysis to allow a leader of 
said strategy modeling team to convince a decision maker to implement a strategy 
favored by said analysis. 

25 

30. The apparatus of Claim 28, said strategy situation analysis module further 
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comprising: 

means for identifying the values of the organization; and 
means for ensuring that the right decisions and strategies considered in an 
analysis. 

31. The apparatus of Claim 28, said data request and reception module further 
comprising: 

means for designing and executing logistics of specifying, acquiring, and 
loading data required for decision and strategy modeling. 

32. The apparatus of Claim 28, said data transformation and cleansing module 
comprising: 

means for verifying, cleansing, and transforming data. 

33. The apparatus of Claim 28, said decision key and intermediate variable 
creation further comprising: 

means for computing intermediate variables from said data, said intermediate 
variables dependent on decision keys; and 

means for constructing a data dictionary. 

34. The apparatus of Claim 28, said data exploration module further comprising: 
means for providing insight into said data by determining which decision keys 

are most relevant for predicting said intermediate variables; and 

means for gaining insight into a customer's business and business processes. 

35. The apparatus of Claim 28, further comprising: 
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means for said decision model structuring module formalizing relationships 
between decisions, decision keys, intermediate variables, and value by connecting 
such in a model. 

36. The apparatus of Claim 28, further comprising: 

means for said decision model quantification module encoding information 
into a decision model. 

37. The apparatus of Claim 28, further comprising: 

means for said strategy creation module applying optimization methods to a 
decision model to determine an optimal strategy for a set of cases. 

38. The apparatus of Claim 28, further comprising: 

means for said strategy creation module evolving using results from a 
decision model being enriched and from strategies tested. 

39. The apparatus of Claim 28, further comprising: 

means for said strategy testing module: 

providing means for evaluating each strategy based on simulation; and 
providing means for evaluating a strategy in the field. 

40. The apparatus of Claim 28, further comprising: 

means for beginning with a simplified value model having less than eight 

drivers 

wherein each of said drivers is modeled crudely by one or two decision keys; 
means for initially including no constraints; 
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means for using said simplified value model for beginning said strategy 
creation module and said strategy testing module, said strategy creation module and 
said strategy testing module indicating areas of said decision model where 
refinement adds particular value; and 

means for after interaction between said decision model and strategies is 
acceptable, iteratively adding details reflecting limitations of a business process. 

41. The apparatus of Claim 28, wherein said team development module comprises: 

a team creation component; and 
a decision quality component. 

42. The apparatus of Claim 28, further comprising: 

means for providing a decision quality process for enabling an organization to 
systematically identify, understand, and track views of quality of decision making. 

43. The apparatus of Claim 90, further comprising: 

means for providing any of six dimensions associated with any of six links in a 
decision quality chain, said six links comprising: 
appropriate frame; 
creative-feasible alternatives; 
meaningful-reliable Information; 
clear values and tradeoffs; 
logically-correct reasoning; and 
commitment to action; 
wherein said chain supports an organization's value. 
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44. The apparatus of Claim 28, said means for providing a strategy situation analysis 
module further comprises: 

means for framing a problem by: 
identifying issues; 
5 developing a decision hierarchy; 

understanding an organization's values; and 
brainstorming and clarifying alternatives; 
means for further understanding said organization's values by 
developing value metrics and prototyping metric results; and 
10 means for planning for data acquisition by: 

identifying intermediate variables; and 
developing a plan for assessment; 
wherein for clarification: 

optional means for returning to said framing a problem step after said 
15 further understanding said organization's values step; and 

optional means for returning to said further understanding said 
organization's values step after said planning for data acquisition step. 



45. The apparatus of Claim 28, said data request and reception module further 
20 comprising: 

means for developing data parameters, comprising any of: 
determining data elements; 
designing a performance period; 
determining data records; and 
25 constructing an initial data dictionary; 

means for determining transfer parameters, comprising: 
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determining transfer format; and 

determining transfer method; 
means for preparing data, comprising: 

assembling transfer data; and 

transferring data; and 
means for loading data on a target system. 

46. The apparatus of Claim 28, said means for providing a data transformation and 
cleansing module further comprising: 

means for validating original data sets, comprising: 
investigating original data sets; and 
cleaning original data sets; 
means for creating analysis data sets, comprising; and 
transforming data; and 
computing additional variables; 
means for validating analysis data sets, comprising; 
transforming data; and 
computing additional variables; 
wherein while creating analysis data sets and problems are uncovered in 
original data sets, then original data sets are further cleaned and retransformed; and 
wherein while validating analysis data sets and problems in said 
transformation, or in original data sets, are uncovered, then such tasks are revisited. 

47. The apparatus of Claim 28, said means for providing a decision key and 
intermediate variable creation module further comprising: 

means for first creating dependent variables useful for decision models, 



210 



comprising: 

identifying concepts; 
triaging concepts; and 
defining dependent variables; and 
5 means for creating independent variables useful for decision models, 

comprising 

identifying concepts; 
triaging concepts; and 
defining dependent variables; 
10 wherein intermediate variables depend on decision keys, other intermediate 

variables, or decisions; and 

wherein each intermediate variable encapsulates a predictive model with a 
dependent variable and independent variables. 

15 48. The apparatus of Claim 28, said means for providing a data exploration module 
further comprising: 

means for applying basic statistical analysis, comprising: 
analyzing continuous variables; and 
analyzing discrete variables; 
20 means for applying variable reduction techniques, comprising: 

applying human and business judgment; and 
applying computational methods; 
means for applying advanced statistical analysis; 
verifying results; and 
25 presenting said results. 
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49. The apparatus of Claim 28, said means for providing a decision model structuring 
module further comprising: 

means for conceptualizing, comprising the steps of: 

selecting intermediate variables that drive value; 
5 building coarse models of intermediate variables; and 

verifying constraints; and 
means for drawing a decision model structure; 

wherein said conceptualizing step is iteratively available for use after said 
drawing step. 

10 

50. The apparatus of Claim 28, said means for providing a decision model 
quantification module further comprising: 

means for modeling intermediate variables; 

means for filling in nodes with models, functions, and/or constants; and 
15 means for validating said decision model; 

wherein said modeling step is iteratively available from said filling in step, and 
wherein said filling in step is iteratively available from said validating said decision 
model step. 



20 51 .The apparatus of Claim 28, further comprising: 

means for providing a score tuner component for automating decision model 
updating and reporting, said score tuner component comprising any of: 

data awareness capability; 

triggering rules; 
25 model history retention; 

self-guided model development; 
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connection to a decision engine; and 
execution and analytic audit trails; 

wherein when a tuning run is triggered, results are reviewed and either 
accepted and an update is deployed, or rejected. 

5 

52. The apparatus of Claim 28, said means for providing a strategy creation module 
further comprising: 

means for performing model optimization, comprising: 

identifying metric variables; 
10 determining optimization parameters; and 

running optimization; 
means for analyzing optimization results, comprising 

viewing optimization results; and 

performing sensitivity analysis on constraints; and 
15 means for developing strategies, comprising: , 

building strategies; and 

refining strategies; 

wherein the performing model optimization step and the analyzing 
optimization results step are available to be used iteratively from either the analyzing 
20 optimization results step or the developing strategies step. 

53. The apparatus of Claim 28, further comprising: 

a non-linear constrained optimization tool for improving test designs and 
optimizing strategies. 

25 
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54. The apparatus of Claim 28, said means for providing a strategy testing module 
further comprising: 

testing strategies, comprising: 

performing strategy simulation; and 
5 performing field testing; and 

evaluating strategies; and 
performing active data collection; 

wherein said testing strategies step is available for being used iteratively from 
said evaluating strategies step. 

10 

55. An apparatus for automating decision model updating and reporting, comprising: 

at least one tuning apparatus, comprising any of: 
data awareness capability; 
triggering rules; 
15 model history retention; 

self-guided model development; 
connection to a decision engine; and 

means for triggering a parameter tuning run execution and analytic audit 

trails; and 

20 means for reviewing results, wherein said results are either accepted and 

an update is deployed, or rejected. 



56. The apparatus of Claim 55, further comprising: 

means for interacting with a server that handles tuning parameters, and 
25 running a scripted model optimization engine for generating new models and 
evaluation reports; 
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wherein said tuning parameters are any of sample sizes, population definition, 
and whether tuning is manually initiated or triggered on a set schedule. 

57. A decisioning client apparatus, comprising: 

5 a decisioning client application processing system for: 

supplying data associated with a customer to a decision engine; and 
requesting a decision; and 

wherein said decision engine comprises a score generation module; 
means for said decision engine, using said score generation module, 
10 generating needed transformations of said data and generating at least one score, 
said at least one score based on at least one score weight of at least one scorecard 
at a time; 

means for said decision engine applying pre-specified decision rules and 
strategies using said data and said transformed data, and at least one score for 
15 generating a vector of recommended decision actions; 

means for said decision engine returning requested data, said transformed 
data, said at least one score, information about said at least one scorecard, and said 
recommended actions to said decisioning client application processing system; 

means for said decisioning client application processing system optionally 
20 implementing said recommended actions, and storing results into a data store. 

58. The decisioning client apparatus of Claim 57, further comprising any of: 

means for said decisioning client application processing system optionally 
taking additional non-score-based decisions over time; 
25 means for said decisioning client application processing system monitoring 

and recording periodic signals from customers and general environment; 
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means for said decisioning client application processing system gathering 
data over time about a customer for helping determine one or more outcomes of 
interest; and 

an asynchronous process periodically triggering preparation of a matched 
5 data set from information about a customer, said information from a predetermined 
time, wherein said results are appended to a growing store of predictive plus 
performance data records; and 

said asynchronous process further comprising means for a score tuner 
component having a triggering mechanism, using said triggering mechanism for 
10 periodically taking said matched data set and producing, if appropriate, score weight 
updates of at least one active scorecard, wherein said scorecard is installed into said 
score generation module after a review. 



59. A score tuner method, comprising the steps of: 
15 providing a score tuning broker module for performing administrative tasks 

associated with updating of score weights, said score tuning broker module 
comprising the steps of: 

determining which scorecards are candidates for tuning; 

checking any operating scorecards are flagged for updates; and 
20 at a pre-specified and parameterized time frequency, determining from 

a rule database which scorecards are up for score weight re-tuning; 

extracting needed data set sub-population based on rules determining 
what sampling window and stratification a current scorecard needs; 

for a scorecard that is a candidate for re-tuning for the current time stamp: 
25 requesting generation of a data set to be used for said tuning; and 

determining what score weight engine project is associated with said 
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scorecard; 

passing a reference to said data set and a project id to said score weight 
engine, and requesting metrics of scorecard performance from said score weight 
engine; and 

5 determining whether updated version is better or not; and 

providing a score weight engine module for performing activities related to 
scorecard results and score weights, said score weight engine module comprising 
the steps of: 

reporting on an existing scorecard's development measures; 
10 computing a scorecard's performance measures on a new sample; 

auditing new predictive data set to ensure that settings are adequate to 
cover data values encountered in said new data; 

creating a new scorecard version of said scorecard being tuned; 

converting raw records in said new predictive data set into coarse classed 
15 records needed for building weights; 

building and scaling score weights of said newly created scorecard given 
said new predictive data; and 

archiving said newly built scorecard and its performance measures. 



20 60. The score tuner method of Claim 59, wherein said score weight engine module is 
script-driven. 



61 .A score tuner method, comprising the steps of: 

providing rapid weights tuning for modifying score weights of a scorecard; 

25 and/or 



217 



providing rapid score alignment for aligning parameters of said scorecard; 
wherein said underlying structure of said scorecard's data is not different from 
original implementation definition. 

5 62. The score tuner method of Claim 61 , further comprising any of the steps of: 

providing a range from a null set of weights to automated and intelligent 
variable selection, classing, model building, scaling, and evaluation; 

providing automated validation of newly developed weights for a fixed set of 
characteristics against a set of previously developed weights on said same 
10 characteristic set; and 

providing automatic re-alignment of a set of scorecards to scale to a previous 
set of odds. 



63. The score tuner method of Claim 61 , further comprising any of the steps of: 
15 providing capability of specifying updating or re-scaling of many models at 

once; and 

providing capability of specifying a schedule for automatic scorecard updates 
and scaling, implying integration into current decision support systems. 

20 64. The score tuner method of Claim 61 , further comprising the step of: 
providing modeling functionality comprising the steps of: 

importing of existing scorecards from decision support software; 
auditing for legal values for scorecard characteristics in a new data set; 
generating summarized data in preparation of the tuning process 

25 including: 

classing of values of data records variables into those expected by the 
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scorecard characteristics; 

generating all summarization needed to run proprietary algorithms from 
a newly provided predictive data set and previously summarized results from past 
tuning runs; and 

5 displaying some summary statistics of records encountered; 

providing specification of expected scaling parameters; 

running an algorithm to generate new score weights for scorecard 
characteristics; 

running evaluation procedures on newly tuned weights; 
10 displaying a scorecard and its evaluation results; 

fitting of log of odds vs. score to determine expected odds by score; 

adjusting alignment parameters to match user supplied expectation; 

exporting of said tuned alignment parameters in a format acceptable to 
decision support software, and while maintaining version control for said scorecard; 
15 and 

providing ability to sequence any of above mentioned steps. 

65. The score tuner method of Claim 61 , further comprising the step of: 

providing reporting and visualization capabilities, comprising summarized 
20 views of new score variable and scorecard characteristics; wherein each view 
includes a comparison of old weights versus new, if applicable, and wherein data is 
divided by defined bins of scorecard characteristics. 

66. A score tuner apparatus, comprising: 

25 a database manager component for managing collection of cases used in 

analysis, and for providing a bridge to multiple possible input data files and/or 
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database management systems; 

a data manager component for providing data records to other data analysis 
components, one case at a time in the event that said data analysis components are 
processing cases in a sample point loop, for exposing a data dictionary to other 
5 components, and for allowing posting variables generated in said data analysis 
components back to said database manager for future recall; 

a modeler component for providing score weight re-optimization and for 
logging odds to score alignment functionality; 

a report collection component for providing viewing, printing, and limited 
10 editing of a standard set of model evaluation reports generated by said modeler; 

a workflow controller for controlling flow of multiple business components 
performing a set of actions that are implied by user specifications and eventually 
fulfilling desired data preparation, analysis, and/or presentation steps; and 

an intelligence agent for performing background checks on results from user 
15 actions and for providing suggestions if a query against its rule base returns a 
recommended intelligent action to take. 

67. The score tuner apparatus of Claim 66, wherein said intelligence agent 
comprises: 

20 means for guiding specification of analytic steps; 

means for reacting to interactive analytic actions with suggestions, via agents, 
for possible changes; and 

means for automating intelligence-assisted decision-making in a sequence of 
analytic actions. 

25 
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68. A system for estimating an uncertainty interval around at least one estimate of at 
least one expected outcome, comprising: 

an input device operable to allow entering and transferring input data to a 

processor; 

5 an output device for displaying human readable results of manipulation of said 

input data; 

one or more communications buses between said input device and said 
processor and said output device and said processor, respectively; and 

said processor comprising a memory, wherein said memory stores at least 
10 one program for quantifying said uncertainty interval due to variation based on case- 
level variation, model variation, and portfolio composition, said program performing a 
sequence of instructions, the sequences of instructions, which, when executed by 
said processor, cause the processor to perform the steps of: 

causing a decision model to encapsulate case-level variation; 
15 implementing non-parametric bootstrapping techniques to capture model 

variation; 

using analysis of historic data on holdout samples to describe case-level error 
distributions; and 

capturing portfolio composition variation as an integral element of said 
20 quantifying said uncertainty interval process. 

69. The system of Claim 68, wherein said process of quantifying said uncertainty 
interval comprises two stages: 

wherein said first stage is repeated for each component model making up said 
25 decision model, resulting in estimating all necessary parameters, and 
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wherein said second stage uses said estimated parameters for rolling said up 
variations into aggregated measures and presenting a range of said at least one 
expected outcome. 



70. A method for estimating an uncertainty interval around at least one estimate of at 
least one expected outcome, comprising the steps of: 

providing an input device operable to allow entering and transferring input 

data to a processor; 

providing an output device for displaying human readable results of 

manipulation of said input data; 

providing communications buses between said input device and said 
processor and said output device and said processor, respectively; and 

said processor comprising a memory, wherein said memory stores at least 
one program for quantifying said uncertainty interval due to variation based on case- 
level variation, model variation, and portfolio composition, said program performing a 
sequence of instructions, the sequences of instructions, which, when executed by 
said processor, cause the processor to perform the steps of: 

providing a decision model to encapsulate case-level variation; 

implementing non-parametric bootstrapping techniques to capture model 
variation; 

using analysis of historic data on holdout samples to describe case-level error 
distributions; and 

capturing portfolio composition variation as an integral element of said 
quantifying said uncertainty interval process. 
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71. The method of Claim 70, wherein said process of quantifying said uncertainty 
interval comprises two stages: 

wherein said first stage is repeated for each component model making up said 
decision model, resulting in estimating all necessary parameters, and 
5 wherein said second stage uses said estimated parameters for rolling said up 

variations into aggregated measures and presenting a range of said at least one 
expected outcome. 
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